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INTRODUCTION 


During  Che  past  few  years  the  military  services  have  been  concerned 
about  the  cost  and  effectiveness  of  their  On-the-Job  Training  (OJT)  pro¬ 
grams.  These  concerns  have  arisen  from  the  recognition  that  in  certain 
types  of  work  situations,  most  notably  those  that  generate  urgent  pressures 
for  production,  requirements  for  OJT  may  conflict  with  production  so  that 
either  production  is  degraded  or,  more  likely,  OJT  is  done  poorly,  if  at 
all.  [  1 J  To  cope  with  such  problems,  some  operating  units  have  formalized 
OJT  by  setting  up  classes  and  laboratories  on  site  to  teach  as  much  of  the 
job  as  possible  in  a  structured  setting  (formal  OJT).  Withdrawal  of  people 
and  equipment  from  operating  units  to  conduct  this  training  has  raised 
questions  about  the  relative  efficiencies  of  the  conventional  and  formal 
approaches.  To  address  problems  such  as  these,  the  Air  Force  is  directing 
a  large  program  of  OJT  research  both  in-  and  out  of -house. 

In  many  situations,  OJT  can  be  the  best  way  to  train  someone  to  pro¬ 
duce  work.  First,  it  can  be  the  most  effective  way.  Since  its  content  and 
conduct  are  usually  dictated  by  the  needs  of  the  workplace,  it  is  immedi¬ 
ately  relevant. [2]  Moreover,  the  trainee  can  be  highly  motivated  to  learn 
because  he  or  she  can  observe  this  relevance  directly.  Finally,  the 
trainer  may  also  be  highly  motivated  to  teach  because  the  results  of  his  or 
her  efforts  may  reduce  his  or  her  own  workload  and  may  contribute  to  posi¬ 
tive  evaluation  by  his  or  her  supervisor,  whereas  inadequate  teaching  may 


(1]  See  Stephenson  and  Burkett,  1975,  and  Carpenter-Huf fman ,  1979,  for 
discussions  of  OJT  deficiencies  and  their  sources. 

[2j  The  content  and  conduct  of  some  OJT  may  not  be  dictated  by  the 
immediate  needs  of  the  workplace,  however;  such  training  is  often 
neglected,  for  obvious  reasons. 


fib-. 


-2- 


harm  both  the  trainer  and  the  work. 

On  the  other  hand,  OJT  may  be  ineffective  in  preparing  a  person  for 
work  that  requires  high  level,  cognitive  skills.  Such  skills  are  best 
taught  in  formal,  structured  settings. 

Second,  OJT  can  be  more  efficient  than  other  training  methods  because 
it  uses  resources  that  produce  work  as  well  as  training.  In  fact,  OJT  may 
use  resources  that  have  been  supplied  to  meet  peak  work  loads  but  that 
would  otherwise  be  idle.  As  noted  earlier,  however,  if  work  loads  are  such 
that  resources  are  rarely  idle,  attempts  to  conduct  OJT  may  interfere  with 
work  production. 

Before  proceeding  further,  I  should  make  clear  what  I  mean  by  OJT. 

The  term  is  sometimes  applied  to  such  diverse  activities  as  instruction  on 
specific  tasks  during  work  performance  and  self-study  of  correspondence 
courses.  I  define  OJT  to  be  any  pre-planned  use  of  work  resources  (people, 
equipment,  facilities)  in  the  work  environment  primarily  for  the  purpose  of 
training  someone  to  produce  work.  By  work  resources ,  I  mean  resources  that 
are  usually  used  in  the  workplace  to  produce  work.  Thus  I  exclude 
correspondence  courses  from  the  definition.  I  also  exclude  formal  OJT  when 
it  is  not  conducted  in  the  work  environment. 

OJT  is  a  particular  kind  of  on-the-job  learning,  which  can  occur 
throughout  a  person's  working  life.  I  do  not  include  in  my  definition  of 
OJT  that  on-the-job  learning  that  takes  place  serendipitous ly  during  the 
production  of  work  but  that  is  not  pre-planned.  It  might  be  possible  to 
assess  the  effectiveness  of  on-the-job  learning  by  measuring  the  level  of 
productivity  a  worker  attains  and  to  assess  its  cost  by  measuring  the  time 


umurt  'H1  n.<u'n«  ’’*> 


and  other  resources  the  worker  requires  to  reach  specified  levels  of  pro¬ 
ductivity.  [1] 

,  In  this  paper  I  discuss  the  concepts  of  effectiveness  and  cost  as 
applied  to  OJT.  These  concepts  are  key  to  explorations  of  policy  questions 
such  as:  Is  an  OJT  program  as  productive  as  it  should,  or  can,  be?  How 
can  an  OJT  program  be  made  more  useful?  Is  the  OJT  program  overly  costly? 
Would  it  be  less  expensive  to  provide  OJT  in  some  way  other  than  is  now 
being  done?. 

Measures  of  effectiveness  and  cost  permit  assessment  of  the  relation¬ 
ship  between  what  an  activity  produces  (output)  and  what  is  required  to 
support  that  activity  (input).  That  these  apparently  simple  concepts  are 
not  straightforward,  however,  is  attested  by  common  errors  in  their  appli¬ 
cation,  some  of  which  I  discuss  below. 

OJT  EFFECTIVENESS 

Defining  effectiveness  to  be  a  measure  of  what  an  activity  produces 
leaves  several  problems  unresolved.  These  fall  into  two  main  groups,  the 
problem  of  hierarchies  of  effectiveness  and  the  problem  of  multiple  out¬ 
comes.  I  discuss  the  hierarchy  problem  first. 

Hierarchies  of  Effectiveness 

The  most  proximate  output  of  OJT  is  the  certification  that  a  trainee 
can  perform  a  set  of  specific  tasks.  OJT  performance,  or  efficiency,  would 
be  measured  by  the  number  of  trainees  that  have  been  certified  to  perform  a 
given  set  of  tasks  using  a  particular  set  of  unit  resources. 

[1]  For  research  that  has  addressed  some  aspects  of  this  concept  see 
Weiher,  et  al.,  1971,  Gay  and  Albrecht,  1978,  and  Albrecht,  1979. 
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The  number  of  tasks  for  which  a  trainee  has  been  certified  is  not, 
however,  a  measure  of  OJT  effectiveness  for  that  trainee.  The  primary  pur¬ 
pose  of  OJT  is  to  improve  a  person's  ability  to  perform  work.  The  assump¬ 
tion  is  that  people  who  can  perform  work  better  will  either  produce  more 
work  or  produce  work  of  higher  quality  or  both;  alternatively,  fewer  peo¬ 
ple  will  be  needed  to  produce  work  of  the  same  quality  or  quantity.  Thus, 
the  effectiveness  of  OJT,  that  is  its  effects  on  something  beyond  its  most 
proximate  output,  should  be  assessed  in  terms  of  the  quality  or  quantity  of 
work  produced  by  people  who  have  had  OJT. 

Although  proximate  output  and  effectiveness  may  be  closely  related, 
the  distinction  between  the  two  is  not  trivial.  Even  though  OJT  is  con¬ 
ducted  within  the  work  environment  there  are  always  fundamental  differences 
between  the  OJT  process  and  work.fl]  For  example,  the  set  of  tasks  chosen 
for  OJT  may  be  unimportant  for  the  job.  Or  the  OJT  trainer  may  teach  the 
trainee  to  follow  correct  procedures  that  are  often  bypassed  on  the  job, 
which  may  put  the  trainee  and  supervisor  in  conflict  in  the  work  situation. 
Even  within  the  training  environment  reliable  evaluation  may  be  difficult, 
particularly  when  tasks  requiring  high-level  cognitive  skills  (such  as 
troubleshooting)  are  involved.  The  common  practice  of  having  the  OJT 
trainer  or  the  trainer's  supervisor  certify  the  competence  of  the  trainee 
compounds  these  difficulties.  These  are  some  of  the  many  possible  points 
of  difference  that  argue  against  limiting  assessments  of  OJT  to  evaluation 
of  its  success  in  the  training  environment  in  teaching  the  tasks  in  the  OJT 
syllabus.  Any  evaluation  of  OJT  effectiveness  must,  however,  include  a 


[1]  See  Goldstein,  1978,  for  an  excellent  discussion  of  these  differ¬ 
ences  . 


careful  assessment  of  its  proximate  output  so  that  sources  of  OJT  strengths 
and  weaknesses  may  be  identified. 

To  illustrate  the  pitfalls  of  confusing  proximate  output  with  effec¬ 
tiveness,  several  evaluators  of  formal  training  have  asserted  that  a  train¬ 
ing  procedure  that  allows  students  to  learn  a  given  body  of  material  in 
less  time  than  before  is  more  effective, [ 1]  even  though  the  proximate  out¬ 
put  of  training  is  the  same.  Student  time  in  training  is  an  input  (perhaps 
the  most  important  single  input)  to  training,  not  an  output  of  training. 
Shortened  training  is  merely  more  efficient  training.  The  conceptual  con¬ 
fusion  may  have  helped  obscure  the  real  issue,  namely,  whether  the  reduced 
training  was  as  effective  in  preparing  the  graduate  for  the  job  as  was  the 
original  training. 

Carrying  the  concept  of  effectiveness  further,  we  can  identify  a 
hierarchy  of  effectiveness.  For  example,  if  OJT  is  being  given  to  people 
who  maintain  aircraft,  the  first  level  in  the  hierarchy  of  effectiveness 
would  be  the  amount  or  quality  of  aircraft  maintenance  they  produce  after 
training,  such  as  the  number  of  defective  aircraft  repaired; [2]  the 
second,  an  assessment  of  the  ability  of  the  aircraft  to  accomplish  the 
assigned  mission,  such  as  the  number  of  successful  sorties  generated;  the 
third,  the  ability  of  the  command  to  which  the  aircraft  are  assigned  to 
accomplish  its  mission,  such  as  the  ability  to  blunt  a  counterstrike;  and 
so  on.  Each  higher  level  elevates  the  significance  of  the  OJT  outcome. 

[1]  This  was  the  original  justification  for  the  Advanced  Instructional 
System,  for  example. 

[2]  For  purposes  of  illustration,  these  examples  are  simplistic;  I 
elaborate  them  shortly. 
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Unfortunately,  each  higher  level  also  introduces  influences  on  the 
outcome  from  sources  other  than  OJT  and  thus  obscures  the  effects  of  OJT  on 
the  outcome.  For  example,  the  rate  of  production  of  aircraft  inspections 
may  rise  in  response  to  an  increase  in  sortie  rate,  rather  than  to 
increased  personnel  skills.  Thus  in  this  case  it  would  be  important  to 
evaluate  the  maintenance  process  as  well  as  its  output  to  insure  that 
extraneous  influences  have  been  considered. 

In  sum,  the  first  problem  in  definition  is  to  establish  the  most  sig¬ 
nificant  level  in  the  hierarchy  of  effectiveness  that  would  permit  unambi¬ 
guous  attribution  of  effectiveness  to  the  OJT  activity.  Although  the  exact 
choice  will  usually  depend  on  specifics  of  the  OJT  activity  of  interest, 
every  effort  should  be  made  to  choose  a  level  at  least  one  step  above  mere 
OJT  output  because  of  the  differences  between  the  OJT  and  work  environ¬ 
ments  . 

Multiple  Outcomes 

At  each  level  in  the  hierarchy  of  effectiveness  OJT  may  have  multiple 
outcomes.  Multiple  outcomes  appear  dven  at  the  level  of  proximate  output, 
since  the  trainee  is  usually  certified  on  a  number  of  tasks,  each  of  which 
might  have  a  different  weight  to  reflect  its  importance  in  the  work  situa¬ 
tion.  Weights  could  include  several  aspects  of  importance,  such  as  the 
frequency  with  which  each  task  is  performed.  The  Air  Force's  Occupational 
Measurement  Center  has  developed  techniques  for  establishing  the  relative 
importance  for  an  entire  career  field  of  each  of  a  set  of  tasks; [1]  these 
techniques  might  be  applied  at  the  work  center  level.  Although  such  a  for- 


[1]  See,  for  example,  Christal,  1975. 


mulation  permits  us  to  conceptualize  the  assessment  of  OJT  output,  this 
assessment  should  also  be  informed  by  an  analysis  of  the  OJT  process  for 
full  understanding  of  its  implications. 

If  we  take  performance  of  work  on  the  job  for  measuring  effectiveness, 
we  may  want  to  use  a  similar  weighting  scheme  to  combine  measures  of  dif¬ 
ferent  kinds  of  output.  It  will  usually  be  possible  to  group  specific  out¬ 
puts  into  more  global  products;  for  example,  the  separate  tasks  required 
to  perform  a  scheduled  aircraft  inspection  could  be  combined  into  the 
number  of  inspections  produced.  However,  as  discussed  earlier,  the  more 
global  the  measure,  the  greater  the  possibility  that  activities  other  than 
those  of  concern  may  affect  it. 

There  is  still  another  way  in  which  multiple  outcomes  may  arise, 
namely,  OJT  may  have  more  than  one  purpose.  For  example,  evidence  of  suc¬ 
cessful  completion  of  OJT  is  often  required  to  qualify  the  trainee  for  pro¬ 
motion.  In  such  a  case,  qualification  may  become  a  separate  objective  for 
the  OJT  activity  because  it  generates  rewards  that  are  not  consequences  of 
work  production.  If  workers  must  compete  for  promotion  for  which  qualifi¬ 
cation  is  prerequisite,  pressures  for  qualification  may  degrade  the  effec¬ 
tiveness  of  OJT  in  improving  the  production  of  work. 

In  assessing  the  effectiveness  of  OJT  programs  with  multiple,  and  pos¬ 
sibly  conflicting,  objectives,  care  must  be  taken  to  assess  attainment  of 
each  objective  and  to  judge,  as  best  one  can,  the  relative  importance  of 
each.  It  is  also  essential,  when  making  such  an  assessment,  to  determine 
the  extent  to  which  multiple  objectives  conflict.  It  is  certainly  not 


legitimate  to  use  a  single  measure,  such  as  the  number  of  people  qualified, 
the  measure  of  OJT  output,  as  has  been  done  in  the  past.[l] 

COST 

Throughout  the  following  discussion,  I  use  cost  as  a  surrogate  for  the 
resources  required  to  support  OJT  (the  inputs  to  the  OJT  process).  Usually 
analysis  of  the  cost  of  an  activity  poses  less  difficult  problems  than  does 
analysis  of  its  effectiveness,  even  though  many  questions  must  still  be 
answered.  For  example,  the  analyst  must  decide  which  resources  should  be 
charged  to  the  activity,  how  dollar  costs  should  be  discounted,  and  how 
costs  should  be  categorized.  In  most  cases,  the  choice  of  analytically 
sound  answers  to  these  questions  depends  on  the  policy  issues  being  con¬ 
sidered,  but  procedures  for  making  the  proper  choices  are  well  esta¬ 
blished.  [2] 

For  example,  decisions  about  how  to  make  an  OJT  program  more  efficient 
often  focus  on  features  of  the  program  that  can  be  manipulated  at  the  local 
level,  such  as  taking  better  advantage  of  training  opportunities  arising 
during  normal  work  or  improving  the  teaching  skills  of  OJT  trainers.  The 
efficiency  of  such  alternative  procedures  will  hinge  on  the  efficiency  of 
local  resource  use.  Such  activities  as  over-all  management  of  the  OJT  pro¬ 
gram  at  higher  headquarters  may  be  little  affected,  and  knowledge  of  the 
cost  of  these  activities,  of  little  help  in  making  the  choice. 

The  cost  of  expanding  the  program  to  more  people  will  be  an  "incremen¬ 
tal"  cost,  that  is  the  cost  of  training  additional  people  beyond  those 
that  the  program  is  already  training,  and  the  average  cost  of  training  a 
single  additional  parson,  a  "marginal"  cost.  Just  as  the  cost  of  training 

[1]  Eisele,  et  al.,  1979. 

[2]  See,  for  example,  Fisher,  1970. 
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a  single  person  must  be  thought  of  in  the  context  of  the  total  training 
activity,  it  is  not  possible  to  determine  the  cost  of  teaching  a  single 
task  in  isolation,  as  some  analysts  have  proposed  to  do.  The  cost  of 
teaching  a  single  task  is  probably  best  determined  by  finding  the  differ¬ 
ence  between  the  cost  of  training  when  the  syllabus  includes  that  task  and 
when  it  does  not. 

Comparing  the  cost  and  effectiveness  of  conventional  OJT  with  the  cost, 
and  effectiveness  of  radically  different  ways  of  delivering  training,  such 
as  formal  OJT  at  the  local  level  or  training  in  task  performance  at  a  for¬ 
mal  school,  raises  many  questions.  The  most  obvious  have  to  do  with 
whether  such  different  delivery  modes  produce  the  same  output.  In  many 
cases  they  may  not  because  of  difficulties  of  maintaining  training 
relevance  at  a  physical  and  psychological  distance  from  the  workplace. 

But  even  if  equivalence  of  output  could  be  assumed,  different 
categories  of  cost  should  probably  be  considered  in  making  cost  compari¬ 
sons.  For  example,  if  the  formal  school  has  another  purpose  and  will  con¬ 
tinue  to  operate  whether  or  not  the  training  in  question  is  given  there, 
the  bulk  of  the  school  administration  and  support  may  not  be  affected  and 
should  not  be  included  in  the  cost  comparison.  On  the  other  hand,  most  of 
the  administration  and  support  activities  for  formal  OJT  given  at  the  local 
level  may  be  chargeable  to  the  formal  program. 

These  kinds  of  questions  must  be  addressed  in  any  cost  analysis.  OJT 
presents  special  problems,  however,  which  are  best  illuminated  by  a  discus¬ 
sion  of  the  different  ways  in  which  work  center  activities  may  be  related 
to  trainee  needs.  The  simplest  type  of  activity  from  the  point  of  view  of 
OJT  resource  use  is  the  activity  whose  only  product  is  training.  I  shall 
refer  to  these  as  Type  I  activities.  An  example  of  such  an  activity  is  the 


.14  ,  -  li  i  llirilft  I 
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supervisor's  production  of  the  OJT  plan  for  new  trainees.  Type  I  activi¬ 
ties  often  account  for  a  very  small  part  of  the  resources  involved  in  the 
total  OJT  program. 

Another  kind  of  activity  requires  little  or  no  OJT;  I  refer  to  these 
as  Type  II  activities.  Examples  are  routine  cleanup  of  facilities  or  car¬ 
rying  equipment  to  or  from  the  workplace.  Persons  who  have  already  had 
work  experience  elsewhere  may  be  able  to  perform  many  or  most  work  activi¬ 
ties  without  OJT. 

Type  II  activities  are  important  because  they  contribute  to  the  pro¬ 
duct  of  the  work  unit  without  levying  additional  resource  needs  for  train¬ 
ing.  Moreover,  the  performance  of  Type  II  activities  by  less  valuable  peo¬ 
ple,  such  as  apprentices,  frees  more  experienced  people  to  concentrate  on 
more  demanding  tasks  and  thereby  offsets  some  of  the  cost  of  OJT.  In  fact, 
some  recent  work  on  the  cost  of  medical  education  in  teaching  hospitals 
suggests  that  some  types  of  patient  care  may  be  provided  less  expensively 
in  teaching  than  in  non-teaching  hospitals  because  student  labor  can  be 
substituted  for  the  labor  of  more  costly  physicians .[ 1]  Care  must  be 
taken,  however,  that  use  of  trainees  for  Type  II  activities  does  not 
degrade  their  OJT. 

I  refer  to  activities  in  which  both  work  and  training  are  produced 
simultaneously  with  the  same  resources  as  Type  III  activities.  Often  most 
OJT  resources  are  devoted  to  Type  III  activities,  but  in  this  case  it  is 
not  possible  to  identify  the  cost  of  producing  training  separately  from  the 


[1]  For  a  discussion  of  this  and  the  following  issue,  see  Massell  and 
Williams,  1977. 
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cost  of  producing  work  in  any  but  an  arbitrary  fashion. [1]  Arbitrary  allo¬ 
cations  shed  no  light  on  the  cost  issue  and  are  often  misleading. 

Where  OJT  involves  Type  III  activities,  it  is  not  possible  to  deter¬ 
mine  the  cost  of  OJT,  alone,  because  work  must  be  produced  to  produce  OJT. 

It  is  possible,  however,  to  estimate  both  the  cost  of  resources  strictly 
attributable  to  OJT  and  the  cost  of  producing  both  work  and  OJT;  both  esti¬ 
mates  may  be  needed  for  making  informed  decisions.  The  cost  of  resources 
strictly  attributable  to  OJT  is  the  difference  between  the  resources 
required  by  units  that  perform  OJT  and  the  resources  required  by  units  that 
produce  essentially  the  same  work  but  do  not  perform  OJT.  If  this  differ¬ 
ence  is  positive,  OJT  reduces  unit  work  productivity.  The  cost  of  produc¬ 
ing  both  OJT  and  work  is  the  cost  of  only  those  work  activities  required 
for  OJT. 

Since  the  OJT  trainee  may  offset  some  of  the  cost  of  OJT  by  partici¬ 
pating  in  Type  II  activities  (which  may  produce  work  at  reduced  cost)  as 
well  as  by  participating  in  the  production  of  work  while  being  trained  in 
Type  III  activities,  to  establish  the  effects  on  cost  of  provision  of  OJT, 
determination  of  the  cost  of  all  three  types  of  activities  may  be  required. 
SUMMARY 

The  foregoing  discussion  makes  clear  that  establishing  the  effective¬ 
ness  and  cost  of  OJT  can  be  a  demanding  task.  To  assess  OJT  effectiveness, 
it  is  not  sufficient  to  count  up  the  number  of  trainees  upgraded  or  certi¬ 
fied,  because  of  differences  between  OJT  and  work.  The  effectiveness  of 


[1]  For  a  discussion  of  this  point,  see  Koehler  and  Slighton,  June 
1973,  p.  540. 


-12- 


OJT  should  be  evaluated  at  least  at  the  level  of  trainee's  production  of 
work.  To  assess  the  cost  of  CUT,  it  is  not  sufficient  to  count  up  the 
resources  that  are,  by  some  means,  attributed  solely  to  OJT,  because  of 
the  interactions  between  OJT  and  production.  The  cost  of  OJT  must  be  as¬ 
sessed  within  the  context  of  all  work  production  that  may  be  affected  by 
OJT,  either  negatively  or  positively. 

OJT  is  a  pervasive  activity  in  many  work  situations.  In  some,  it  may 
be  the  best  way  to  prepare  workers  for  their  jobs;  in  others,  different 
approaches  may  be  more  desirable.  At  present,  our  understanding  of  how  to 
identify  these  situations  is  hampered  by  our  ignorance  of  either  the  effec¬ 
tiveness  or  the  cost  of  OJT.  The  increasing  importance  of  skilled  persons 
in  today's  military  services  argues  that  improving  our  ability  to  identify 
the  best  ways  to  go  about  OJT  may  be  worth  the  effort. 

This  is  not  an  impossible  task,  but  it  will  require  sizeable  invest¬ 
ments  of  time,  skill,  and  thoughtfulness.  A  logical  first  step  is  to  con¬ 
struct  a  general  model  of  OJT  and  to  test  its  adequacy  in  describing 
specific  situations. 
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